feat(ci): wanda ray image builds, uploads to Dockerhub #59936

andrew-anyscale · 2026-01-07T15:57:49Z

Adds Dockerfile and Wanda files for Ray images, pulling artifacts from previous steps. Buildkite steps also updated to use these, including both build and upload steps.

Topic: ray-image
Relative: push-script

Signed-off-by: andrew [email protected]

andrew-anyscale · 2026-01-07T15:57:53Z

Reviews in this chain:
└#60197 [ci] Add RayImagePushContext for publishing wanda Ray images
　└#59936 feat(ci): wanda ray image builds, uploads to Dockerhub
　　└#59937 Add wanda anyscale image builds for release tests

andrew-anyscale · 2026-01-07T15:57:53Z

#	head	base	diff	date	summary
0	02a8921d	f104bf59	diff	Jan 7 7:57 AM	9 files changed, 593 insertions(+), 49 deletions(-)
1	fe687866	288cadb6	diff	Jan 7 7:58 AM	0 files changed
2	46cc8625	288cadb6	diff	Jan 7 8:38 AM	4 files changed, 440 insertions(+), 31 deletions(-)
3	70df7561	164e2e4c	rebase	Jan 7 11:29 AM	0 files changed
4	ae32d45a	b4fb3e06	rebase	Jan 7 12:21 PM	0 files changed
5	04cfd26a	cd792539	rebase	Jan 7 17:29 PM	0 files changed
6	55d1473f	7d257e47	rebase	Jan 7 17:31 PM	0 files changed
7	84d74797	e93a821d	diff	Jan 8 8:27 AM	1 file changed, 6 insertions(+), 14 deletions(-)
8	c275fe24	8a4bdf5b	rebase	Jan 8 8:32 AM	0 files changed
9	7c78fa2d	a0b03e94	rebase	Jan 8 12:01 PM	0 files changed
10	29cf5abd	32347fa5	rebase	Jan 8 12:51 PM	0 files changed
11	d566c8a8	8d1057a9	diff	Jan 8 13:16 PM	0 files changed
12	91900297	d8901f73	rebase	Jan 8 13:33 PM	0 files changed
13	73822f53	9e438258	rebase	Jan 8 14:30 PM	0 files changed
14	94805e95	6801f72f	rebase	Jan 9 8:45 AM	0 files changed
15	fc79d67d	9834ea38	rebase	Jan 9 10:01 AM	0 files changed
16	db94f935	25e0d03c	rebase	Jan 9 10:59 AM	0 files changed
17	05c06707	2a361721	rebase	Jan 9 12:36 PM	0 files changed
18	a6718508	b81d4cb4	rebase	Jan 9 13:32 PM	0 files changed
19	51fe33a5	4933e1ce	rebase	Jan 13 8:55 AM	0 files changed
20	ed74c8c7	19d388a4	rebase	Jan 13 10:55 AM	0 files changed
21	6242ab25	7c1782d4	rebase	Jan 13 15:50 PM	0 files changed
22	a941fd54	09f88fb0	rebase	Jan 13 15:50 PM	0 files changed
23	9db9513a	c7f89bff	rebase	Jan 13 17:03 PM	0 files changed
24	6d4a0beb	e2ebabb6	rebase	Jan 13 17:09 PM	0 files changed
25	dd5133b8	7b8a44a4	rebase	Jan 13 17:09 PM	0 files changed
26	ae00964a	1410defe	rebase	Jan 13 17:12 PM	0 files changed
27	021b2e13	64ac2fd8	rebase	Jan 13 17:14 PM	0 files changed
28	52b7b213	d5b11ffd	rebase	Jan 13 17:33 PM	0 files changed
29	8cfa977c	524d02c4	diff	Jan 13 21:52 PM	0 files changed
30	924d46f1	d1f94807	diff	Jan 15 13:10 PM	10 files changed, 476 insertions(+), 432 deletions(-)
31	275827a2	d1f94807	diff	Jan 15 13:13 PM	0 files changed
32	04d68065	545eefc9	diff	Jan 15 13:14 PM	1 file changed, 11 insertions(+)
33	6e985e93	545eefc9	diff	Jan 15 13:40 PM	3 files changed, 13 insertions(+), 15 deletions(-)
34	17d793f4	545eefc9	diff	Jan 15 13:47 PM	1 file changed, 3 insertions(+), 10 deletions(-)
35	716cc651	545eefc9	diff	Jan 15 16:04 PM	9 files changed, 47 insertions(+), 14 deletions(-)
36	976be9fb	545eefc9	diff	Jan 15 16:48 PM	7 files changed, 2 insertions(+), 14 deletions(-)
37	f4e1306c	a0e25707	diff	Jan 15 17:41 PM	0 files changed
38	73da8387	06d411f4	rebase	Jan 15 18:02 PM	0 files changed

gemini-code-assist

Code Review

This pull request successfully migrates the Ray Docker image builds to use wanda, which simplifies the CI configuration. The new wanda.yaml files and the shared Dockerfile are well-structured. However, I've identified a few areas for improvement, primarily in the new push_ray_image.py script. There's a critical issue where the script only supports pushing the main ray images, which is a regression from the previous system that also handled ray-extra and ray-llm variants. Additionally, the script duplicates significant portions of the image tagging logic from another file, which will create maintenance challenges. I've also included a suggestion to reduce duplication in the .buildkite/build.rayci.yml file using YAML anchors.

ci/ray_ci/automation/push_ray_image.py

.buildkite/build.rayci.yml

gemini-code-assist · 2026-01-07T16:01:22Z

ci/ray_ci/automation/push_ray_image.py

+
+# GPU_PLATFORM is the default GPU platform that gets aliased as "gpu"
+# This must match the definition in ci/ray_ci/docker_container.py
+GPU_PLATFORM = "cu12.1.1-cudnn8"


The GPU_PLATFORM constant is hardcoded here and is also defined in ci/ray_ci/docker_container.py. Duplicating constants can lead to inconsistencies if one is updated and the other is not.

This constant should be defined in a single, shared location and imported where needed. This could be done as part of the larger refactoring of the tag generation logic I mentioned in another comment.

ci/ray_ci/automation/push_ray_image.py

andrew-anyscale · 2026-01-15T22:12:37Z

ci/docker/ray-extra-image-cpu.wanda.yaml

@@ -0,0 +1,13 @@
+name: "ray-extra-py$PYTHON_VERSION-cpu$ARCH_SUFFIX"
+disable_caching: true


I chose to disable_caching because the ray-wheel image has its own caching disabled. I need to double check this assumption.

andrew-anyscale · 2026-01-16T00:34:08Z

Results from #60186:

=== Comparing images ===
Image 1: 029272617770.dkr.ecr.us-west-2.amazonaws.com/rayproject/citemp@sha256:5f324c6e299db38c60f1361847ad351788ba9808e0cf5462924bc0469ee01ee7
Image 2: rayproject/ray:nightly.260115.f18213

Fetching image configs...
=== Config differences (excluding history/rootfs) ===
--- /var/folders/dg/jkv044t9525bdy3bq2ksxwjh0000gp/T/tmp.5mtTUMytHA/config1.json	2026-01-15 16:32:23
+++ /var/folders/dg/jkv044t9525bdy3bq2ksxwjh0000gp/T/tmp.5mtTUMytHA/config2.json	2026-01-15 16:32:23
@@ -15,18 +15,17 @@
     ],
     "WorkingDir": "/home/ray",
     "Labels": {
-      "io.ray.ray-commit": "0d911cd230fbcf90963348136f3693df658cd8f2",
+      "io.ray.ray-commit": "f182131f846c8de5eef21ad6a0621aa30b6005cb",
       "io.ray.ray-version": "3.0.0.dev0",
       "org.opencontainers.image.ref.name": "ubuntu",
       "org.opencontainers.image.version": "22.04"
     },
-    "ArgsEscaped": true,
     "OnBuild": null,
     "Shell": [
       "/bin/bash",
       "-c"
     ]
   },
-  "created": "2026-01-16T00:20:02.976635141Z",
+  "created": "2026-01-15T07:05:55.735088841Z",
   "os": "linux"
 }

=== Environment variables ===
--- Image 1 ---
--- Image 2 ---
--- Diff ---

=== Labels ===
--- Image 1 ---
{
  "io.ray.ray-commit": "0d911cd230fbcf90963348136f3693df658cd8f2",
  "io.ray.ray-version": "3.0.0.dev0",
  "org.opencontainers.image.ref.name": "ubuntu",
  "org.opencontainers.image.version": "22.04"
}
--- Image 2 ---
{
  "io.ray.ray-commit": "f182131f846c8de5eef21ad6a0621aa30b6005cb",
  "io.ray.ray-version": "3.0.0.dev0",
  "org.opencontainers.image.ref.name": "ubuntu",
  "org.opencontainers.image.version": "22.04"
}
--- Diff ---
--- /var/folders/dg/jkv044t9525bdy3bq2ksxwjh0000gp/T/tmp.5mtTUMytHA/labels1.json	2026-01-15 16:32:25
+++ /var/folders/dg/jkv044t9525bdy3bq2ksxwjh0000gp/T/tmp.5mtTUMytHA/labels2.json	2026-01-15 16:32:26
@@ -1,5 +1,5 @@
 {
-  "io.ray.ray-commit": "0d911cd230fbcf90963348136f3693df658cd8f2",
+  "io.ray.ray-commit": "f182131f846c8de5eef21ad6a0621aa30b6005cb",
   "io.ray.ray-version": "3.0.0.dev0",
   "org.opencontainers.image.ref.name": "ubuntu",
   "org.opencontainers.image.version": "22.04"

=== Entrypoint/Cmd ===
--- Image 1 ---
{
  "Entrypoint": null,
  "Cmd": [
    "/bin/bash"
  ]
}
--- Image 2 ---
{
  "Entrypoint": null,
  "Cmd": [
    "/bin/bash"
  ]
}

=== Working Dir ===
--- Image 1 ---
/home/ray
--- Image 2 ---
/home/ray

=== User ===
--- Image 1 ---
1000
--- Image 2 ---
1000

=== Exposed Ports ===
--- Image 1 ---
{}
--- Image 2 ---
{}

=== Architecture/OS ===
--- Image 1 ---
{
  "architecture": "amd64",
  "os": "linux"
}
--- Image 2 ---
{
  "architecture": "amd64",
  "os": "linux"
}

=== Important Files ===
Pip freeze diff summary (packages that differ):
< ray @ file:///home/ray/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#sha256=bc4f590d0b8248746ffd20b5da7348f294fa508dadac8188221df0318d9551bd
> ray @ file:///home/ray/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#sha256=680cc52fb4116ef3565a1d1df5975228672965d57d355792cac767c48473ffef
(no differences)

--- Ray wheel location in pip-freeze ---
Image 1:
ray @ file:///home/ray/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#sha256=bc4f590d0b8248746ffd20b5da7348f294fa508dadac8188221df0318d9551bd
Image 2:
ray @ file:///home/ray/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#sha256=680cc52fb4116ef3565a1d1df5975228672965d57d355792cac767c48473ffef

andrew-anyscale · 2026-01-16T00:38:35Z

.buildkite/build.rayci.yml

+    env:
+      PYTHON_VERSION: "{{matrix}}"
+      ARCH_SUFFIX: ""
+      RAY_VERSION: "3.0.0.dev0"


This will be deprecated in favor of an envfile--

leaning towards having an env_file support in wanda spec/config that can read in build args to set, so that we can do labeling in the dockerfile that are based on file contents

Adds Dockerfile and Wanda files for Ray images, pulling artifacts from previous steps. Buildkite steps also updated to use these, including both build and upload steps. Topic: ray-image Relative: push-script Signed-off-by: andrew <[email protected]>

cursor · 2026-01-16T02:14:25Z

ci/docker/ray-image.Dockerfile

+    echo "Error: Expected 1 ray wheel file, but found ${#WHEEL_FILES[@]} in /home/ray/." >&2
+    ls -l /home/ray/*.whl >&2
+    exit 1
+fi


Glob validation fails to detect zero wheel files

Low Severity

The wheel file validation logic uses a bash glob pattern without setting nullglob, so if no files match /home/ray/ray-*.whl, the array contains the unexpanded literal pattern string as one element. The check ${#WHEEL_FILES[@]} -ne 1 then incorrectly passes, and the error message would claim "found 1" wheel file when actually there are zero. The pip install would still fail eventually, but with a confusing "file not found" error rather than the intended descriptive message.

This was referenced Jan 7, 2026

Migrate wheel builds to wanda (x86_64) #59935

Merged

Add wanda anyscale image builds for release tests #59937

Draft

Add wanda builds for aarch64 architecture #59938

Draft

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 02a8921 to fe68786 Compare January 7, 2026 15:59

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from f104bf5 to 288cadb Compare January 7, 2026 15:59

gemini-code-assist bot reviewed Jan 7, 2026

View reviewed changes

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from fe68786 to 46cc862 Compare January 7, 2026 16:38

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from 288cadb to 164e2e4 Compare January 7, 2026 19:29

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 46cc862 to 70df756 Compare January 7, 2026 19:29

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from 164e2e4 to b4fb3e0 Compare January 7, 2026 20:21

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 70df756 to ae32d45 Compare January 7, 2026 20:21

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from b4fb3e0 to cd79253 Compare January 8, 2026 01:29

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from ae32d45 to 04cfd26 Compare January 8, 2026 01:29

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from cd79253 to 7d257e4 Compare January 8, 2026 01:31

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 04cfd26 to 55d1473 Compare January 8, 2026 01:31

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from 7d257e4 to e93a821 Compare January 8, 2026 16:27

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 55d1473 to 84d7479 Compare January 8, 2026 16:27

andrew-anyscale mentioned this pull request Jan 8, 2026

wanda python-agnostic ray-cpp wheel (py3-none tag)(x86_64) #59969

Merged

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 84d7479 to c275fe2 Compare January 8, 2026 16:32

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch 2 times, most recently from 8a4bdf5 to a0b03e9 Compare January 8, 2026 20:01

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from c275fe2 to 7c78fa2 Compare January 8, 2026 20:01

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from a0b03e9 to 32347fa Compare January 8, 2026 20:52

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 7c78fa2 to 29cf5ab Compare January 8, 2026 20:52

andrew-anyscale changed the base branch from andrew/revup/master/ray-wheel to andrew/revup/master/ray-cpp-wheel January 8, 2026 21:16

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 29cf5ab to d566c8a Compare January 8, 2026 21:16

andrew-anyscale force-pushed the andrew/revup/master/ray-cpp-wheel branch from 8d1057a to d8901f7 Compare January 8, 2026 21:33

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from d566c8a to 9190029 Compare January 8, 2026 21:33

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch 2 times, most recently from 021b2e1 to 52b7b21 Compare January 14, 2026 01:33

andrew-anyscale force-pushed the andrew/revup/master/ray-cpp-wheel branch from 64ac2fd to d5b11ff Compare January 14, 2026 01:33

Base automatically changed from andrew/revup/master/ray-cpp-wheel to master January 14, 2026 03:29

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch 2 times, most recently from 8cfa977 to 924d46f Compare January 15, 2026 21:10

andrew-anyscale marked this pull request as ready for review January 15, 2026 21:10

andrew-anyscale requested a review from a team as a code owner January 15, 2026 21:10

andrew-anyscale changed the title ~~Add wanda ray image builds for Docker Hub~~ feat(ci): wanda ray image builds, uploads to Dockerhub Jan 15, 2026

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch 2 times, most recently from 275827a to 04d6806 Compare January 15, 2026 21:14

cursor bot reviewed Jan 15, 2026

View reviewed changes

ci/ray_ci/automation/push_ray_image.py Outdated Show resolved Hide resolved

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 04d6806 to 6e985e9 Compare January 15, 2026 21:40

andrew-anyscale mentioned this pull request Jan 15, 2026

DNM: A/B validating Wanda-generated Ray images #60186

Draft

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 6e985e9 to 17d793f Compare January 15, 2026 21:47

andrew-anyscale marked this pull request as draft January 15, 2026 22:03

andrew-anyscale commented Jan 15, 2026

View reviewed changes

andrew-anyscale marked this pull request as ready for review January 16, 2026 00:04

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 17d793f to 716cc65 Compare January 16, 2026 00:04

andrew-anyscale commented Jan 16, 2026

View reviewed changes

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 716cc65 to 976be9f Compare January 16, 2026 00:48

ray-gardener bot added the devprod label Jan 16, 2026

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 976be9f to f4e1306 Compare January 16, 2026 01:41

andrew-anyscale changed the base branch from master to andrew/revup/master/push-script January 16, 2026 01:41

andrew-anyscale mentioned this pull request Jan 16, 2026

[ci] Add RayImagePushContext for publishing wanda Ray images #60197

Open

andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from f4e1306 to 73da838 Compare January 16, 2026 02:02

andrew-anyscale force-pushed the andrew/revup/master/push-script branch from a0e2570 to 06d411f Compare January 16, 2026 02:02

cursor bot reviewed Jan 16, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ci): wanda ray image builds, uploads to Dockerhub #59936

feat(ci): wanda ray image builds, uploads to Dockerhub #59936

andrew-anyscale commented Jan 7, 2026 •

edited

Loading

Uh oh!

andrew-anyscale commented Jan 7, 2026 •

edited

Loading

Uh oh!

andrew-anyscale commented Jan 7, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Jan 7, 2026

Uh oh!

Uh oh!

andrew-anyscale Jan 15, 2026

Uh oh!

andrew-anyscale commented Jan 16, 2026 •

edited

Loading

Uh oh!

andrew-anyscale Jan 16, 2026

Uh oh!

cursor bot Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,13 @@
		name: "ray-extra-py$PYTHON_VERSION-cpu$ARCH_SUFFIX"
		disable_caching: true

feat(ci): wanda ray image builds, uploads to Dockerhub #59936

Are you sure you want to change the base?

feat(ci): wanda ray image builds, uploads to Dockerhub #59936

Conversation

andrew-anyscale commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrew-anyscale commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrew-anyscale commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrew-anyscale Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

andrew-anyscale commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrew-anyscale Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor bot Jan 16, 2026

Choose a reason for hiding this comment

Glob validation fails to detect zero wheel files

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andrew-anyscale commented Jan 7, 2026 •

edited

Loading

andrew-anyscale commented Jan 7, 2026 •

edited

Loading

andrew-anyscale commented Jan 7, 2026 •

edited

Loading

andrew-anyscale commented Jan 16, 2026 •

edited

Loading